Data integration for distributed and massively parallel processing environments

ABSTRACT

Methods and systems for large scale data integration in distributed or massively parallel environments comprises a development phase wherein the results of a proposed jobflow can be viewed by the user during development, including the results of upstream units where the data sources and data targets can be any of a variety of different platforms, and further comprises the use of remote agents proximate to those data sources and data targets with direct communication between the associated agents under the direction of a topologically central controller to provide, among other things, improved security, reduced latency, reduced bandwidth requirements, and faster throughput.

RELATED APPLICATION

The present application claims priority to U.S. provisional patentapplication No. 62/502,594 filed 5 May 2017, which is incorporatedherein by reference in its entirety.

BACKGROUND Field of the Invention

The present invention relates generally to data integration in eitherdistributed or massively parallel processing environments, and, in oneaspect, more particularly relates to interactive development of extract,load and transform data flows while, in another aspect, relates to theuse of geographically dispersed agents to simplify extract, load andtransform processes with enhanced security and improved datacompression.

Related Art

More and more, data analysts require the use of data outside the controlof their own organizations. Greatly increasing amounts of data availablevia the Web, new technologies for linking data across datasets, and anincreasing need to integrate structured and unstructured data all drivethis trend. While such data is often referred to as “Big Data”, thediversity of sources for these heterogeneous datasets, both structuredand unstructured, lend themselves to the term “Broad Data” in additionto being “Big Data.”

While Big Data applications historically developed either within largecompanies that had significant archives of their own, or the data cameat relatively high data rates and sizes, more recently big dataapplications involve data distributed much more broadly. In someinstances, for example, the data can be the result of a distributedproject such as research conducted at a variety of geographically orlogically disparate locations. Conventional approaches for handling suchdiverse datasets have increasingly involved such research communitiesdeveloping controlled vocabularies and/or ontologies as well as metadatastandards to help with data integration. In other instances, companiesare increasingly looking at how they might integrate their structureddata holdings with those of others and to explore links to both internaland external unstructured data sources. This is especially true fornonprofits, small companies, and others trying to get a benefit from bigdata where access to data outside their own control is even morecrucial.

Having the ability to integrate massive amounts of data available fromdiverse datasets, catalogs, domains, and cultures can provide data userswith the ability to find, access, integrate, and analyze combinations ofdatasets specific to their needs in ways not previously possible.However, traditional database techniques have not generally focused onthe challenges that result from trying to mine data from largerepositories that are not organized for such searching, for linking whatis found to other data, or for reusing and repurposing the data withoutmassive effort. Most “big data” analytics to date have assumed knowndata structures and organized data systems, where the dataset has beencarefully curated and cleaned.

These traditional assumptions simply do not apply to many types ofunstructured data, and those traditional techniques have given way tosearch engines and metadata-based tools. More sophisticated language andmetadata-markup tools are becoming available for searching documentsets. Network and social media analytics that constitute the bulk of“big data” projects at search and social media companies are largelypowered by these technologies, harnessing a combination of languagetools with learning based on the dynamics of users' interactions.

An increasing number of challenges are not amenable to solution by theseconventional techniques. In many instances, significant conflict existsbetween retrieval and precision, both of which relate to relevance, andmetrics based on precision and recall can have different meaningdepending upon the tools and context. This conflict is magnified whenthe objective is to apply data analytics to unstructured or diverselystructured datasets. While sufficient data integration can overcome atleast some of this conflict, conventional approaches to such dataintegration typically result in unworkable complexity and a lack oftransparency that hinders or prevents successful debugging oftransformation logic. The result is that attempts at efficientintegration of large datasets from diverse sources has been largelyunsuccessful.

Further, data integration has typically involved moving large amounts ofdata across relatively long distances. Given the confidential andproprietary nature of such data, these movements have historically runthe risk of exposing confidential information to third parties. Whilevarious encryption techniques have been used, the challenges ofencrypting large data sets for transmission across long distances can bedaunting. Compression techniques have been used in the past, but againthe challenges can become daunting because of the volume of compressiontypically needed and the security risks involved concerning both privacyand confidentiality.

As a result, there has been a long-felt need for systems and methods forlarge-scale data integration that permits the user to understand theimpact of the user's transformation rules and provides transparency in amanner that permits efficient and effective development of job flows,unit flows, and their debugging.

In addition, there has also been a long felt need for systems andmethods for large scale data integration that permit faster, morereliable, and more secure data throughput resulting from one or more ofdistributed, remote processing proximate to the data sources, datacompression, data encryption, and direct communication from remotesources to one or more remote targets to reduce system latency, reducebandwidth requirements, and minimize the exposure of raw data.

Therefore, what is needed is a system and method that overcomes thesesignificant problems found in the conventional systems as describedabove.

SUMMARY

The present invention overcomes many of the limitations of the priorart. In particular, embodiments of the present invention provide a userinterface and supporting data integration structure and process thatenables a user to build set-based transformation rules for integratingdata from a plurality of sources and to understand, from a display ofthe resulting data, the outcome of at least some of the transformationrules as each such rule is applied. Thus, for a data integrationimplementation that involves, for example, three transformations fromtwo sources, the data integration techniques of the present inventioncan allow a display of the resulting data as the first transformationrule is applied, and again as the third transformation rule is applied.If desired, a display is possible upon application of eachtransformation rule. A transformation unit can aggregate a plurality oftransformation rules, and multiple transformation units can be developedto achieve a desired large scale data integration.

Further, the system of the present invention permits the user, whenchanging an upstream transformation rule, to display a preview of thedata results from all dependent or down-stream transformations. Forexample, a modification of the logic on a first transformation unitcauses a change in the data output in a third, downstreamtransformation. In an embodiment, the user modifying one or more rulesof the first transformation unit can display a preview of the dataresulting from the third transformation rule. Further, the volume of asample can be selected, and the data generated for preview can bepreserved in memory or disk, depending on user preference.

In an embodiment of another aspect of the present invention, statelessagents, like the controller a software application, under the control ofone or more central controller components are implemented to assist inextracting, loading and transforming data in a highly distributednetwork of data systems, which include both data sources and datatargets. By distributing the agents proximate to the data sources, forexample on the same server farm as the source or a server farm near thesource, the agents can perform not only extract/load/transformoperations close the source. In addition, agents proximate to the sourcecan also encrypt and compress data in accordance with metadatainstructions established by the user, resulting in substantially lessrisk of exposure of confidential data. Further, agents at a data sourcecan transfer data directly to agents at a data target, including, insome embodiments, compression and encryption, rather than having to passthe data through a central processing point. These benefits result inincreased speed and efficiency for the overall data integration schemewithout requiring that all data be kept in a single repository. Themetadata rules can be maintained in any convenient location accessibleby one or more controller components. Multiple agents can be implementedwithin a single server farm to service a single large data source, topermit load balancing for further increased efficiency.

Other features and advantages of the present invention will become morereadily apparent to those of ordinary skill in the art after reviewingthe following detailed description and accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

The structure and operation of the present invention will be understoodfrom a review of the following detailed description and the accompanyingdrawings in which like reference numerals refer to like parts and inwhich:

FIG. 1A illustrates the data integration environment in which thepresent invention operates.

FIG. 1B illustrates at a high level the interaction of a distributednetwork of agents under the management of one or more controllers inaccordance with an embodiment of the present invention.

FIG. 1C illustrates in block diagram form a variety of types of datasources that can be accessed in at least some embodiments of the presentinvention.

FIG. 1D illustrates in block diagram form the interrelationships of thesoftware modules of an embodiment of the data integration suite of thepresent invention.

FIG. 2 illustrates in block diagram form the hardware and networkstructure of an embodiment of the data integration system of the presentinvention.

FIGS. 3A-3B illustrate in block diagram form an embodiment of thesoftware components of the data integration suite of an aspect of thepresent invention.

FIG. 4 illustrates an embodiment of a metadata design state diagram ofdata point management in accordance with the present invention.

FIG. 5 illustrates an embodiment of a metadata design state diagram ofdata object management in accordance with the present invention.

FIG. 6 illustrates an embodiment of a metadata design state diagram ofdata flow management in accordance with the present invention.

FIG. 7 illustrates in state diagram form an exemplary data flow processfor computing the data for an active transformation, generating a resultset, and presenting the result to a browser data window.

FIGS. 8A-8B illustrate an interactive design state diagram illustratingdata flow among the various modules in accordance with the presentinvention.

FIG. 9 illustrates an embodiment of a metadata design state diagram ofjob flow in accordance with the invention.

FIGS. 10A-10C illustrate an embodiment of an execution state diagram forjob flow in accordance with the invention.

FIG. 11 illustrates in block diagram form the interrelationship of themodules of controller and agent in accordance with an embodiment of thepresent invention.

FIGS. 12A-12B illustrate a controller and agent state diagram inaccordance with an embodiment an aspect of the present invention.

FIG. 13 illustrates an agent management console component diagram inaccordance with an embodiment of the present invention.

FIGS. 14A-14B illustrate in state diagram form an embodiment of theinteractions of the controller and an agent group.

FIG. 15 is a block diagram illustrating an example wired or wirelessprocessor enabled device that may be used in connection with variousembodiments described herein.

In the following description, like elements will in at least some casesbe described with the same reference numerals for multiple figures forthe sake of consistency and clarity.

DETAILED DESCRIPTION

Referring first to FIG. 1A, an embodiment of the environment 100 inwhich the present data integration invention operates can be betterappreciated. As will be appreciated in greater detail hereinafter, dataintegration in accordance with the present invention comprises tworelated aspects: in a first aspect, a job flow must be developed basedon the particular data which a user seeks to integrate, taking intoaccount the sources of the data, their formats, and their geographicallocation, among other things. The development of a job flow involvesdevelopment of a data flow for each such source, typically involving oneor more extract/load/transform (sometimes “E/L/T” hereinafter)functions, together with any necessary E/L/T functions appropriate tomove the data or results to a target. Then, in a second aspect,following the development of a job flow, the data integration job mustexecute efficiently, taking into account appropriate security, audit,and other data transfer concerns. Each of these aspects operates underthe control of a data integration suite, shown generally at 105 in FIG.1A. The data integration suite 105 comprises a user interface 110, acontroller 115, a command line interface 120, a metadata/rules engine125, and one or more agents, indicated at 130, 135 and 140. Under thedirection of the controller 115 and based on the metadata in the engine125, one or more first agents, indicated at 130, perform specifiedextract, load and/or transform functions on the data selected from anyof several sources 145. In some embodiments, one or more other agents,indicated at 135, directs to a processing engine 150 the transfer of allor a portion of the data or the results received from the agents. Theprocessing engine is typically user-designated, and is typically amassively parallel processing platform such as Hive, Spark, Impala,BigSQL, Netezza, Teradata, Hawq, or similar. The processing engineperforms further user-specified transforms as specified by metadatainstructions maintained in the metadata repository 125, after which theresulting data can be provisioned by one or more agents 140 for use bydownstream applications 155 such an analytics 160 or other downstreamsystems 165.

Referring next to FIG. 1B, the general interrelationships of thesoftware modules of an embodiment of the present invention can be betterappreciated. In particular, the modules of an embodiment of the dataintegration suite 105 can be seen to comprise a user interface 110 whichcommunicates bi-directionally with a controller module 115 configured asan application server. Depending upon the implementation, the userinterface 110 can comprise a browser with a user interface written inany convenient manner, such as with HTML5, AngularJS, CSS 3.0,Bootstrap, and so on. During the data integration development phase, theuser interface allows the user to define or edit the specific E/L/Tfunctions that form the units of a job flow and to test the data flow asit is being developed, including the ability to see a visual display ofthe results. During the execution phase, the user interface provides theuser the ability to login and select for execution a previously storedjob flow. The controller module 115 can be written in any suitablelanguage, such as Java 8, Hibernate 4, Spring 4, Tomcat, RESTful,Node.js, ActiveMQ, or CentOS/RHEL/Linus, among others.

The controller module 115 also communicates with a command lineinterface (“CLI”) module 120 in a conventional manner. The CLI can bewritten in any convenient language, such as a Linux shell, RESTful, Java8 or MS-DOS. In addition, the controller module 115 communicates withone or more Agent Module 130, 135, 140 and a Metadata/Rules EngineModule 125. The Agent Module(s) 130, 135, 140 are each an applicationand respond to metadata instructions from the Controller Module toperform Extract/Load/Transform functions, and can be written in anyconvenient language such as Java 8, Spring 4, or Windows/Linux. TheMetadata/Rules Engine module 125 is essentially a database that servesas a repository for storing the metadata instructions that form a jobflow and its component data flows, as well as various types of systemdata, as explained further in connection with FIG. 3. The metadata/rulesengine 125 can be written in, for example, PostgreSQL 9.x or Oracle 1 gor other convenient language.

FIG. 1C shows in block diagram form the relationship among thecomponents of the data integration suite together with the network ofdiverse data systems, indicated generally at 145, where the distributeddeployment of a plurality of agents 170-190 permits efficient E/L/Tfunctions to be performed. As will be appreciated in greater detailhereinafter, the agents can be distributed over a wide geographic area,such as multiple locations around the globe, and are typicallyimplemented in a server reasonably proximate to a data source or target,which helps to provide improved security and to reduce the volume ofdata that needs to be transported for further integration or subsequentanalytics. During the development of a job flow using the dataintegration suite described hereinafter, the controller 115 stores inrepository 125 metadata instructions for extract/load/transform actionsto be performed by the agents associated with each of the data sources,as more fully seen in FIG. 1D et seq. During execution of a job flow,the controller retrieves those user-defined metadata instructions fromrepository 125 in accordance with which E/L/T instructions are issued toagents 170-190. In an embodiment, the agents are configurable to performany of a plurality of functions, including the afore-mentioned extract,load and transform tasks. The agents are configured to communicate toany of the associated data source/target, the controller, or directly toanother agent, as defined by the metadata instructions that comprise thejob flow. For security, in at least some embodiments the metadataincludes instructions to encrypt and compress the data and also to auditthe transfer of the process. The controller also manages theconfiguration of each agent as well as monitoring each agents‘heartbeat’.

FIG. 1D illustrates in greater detail a network diagram showing thedistributed nature of the agents and data sources/targets with respectto the controller, the associated user interface, and the metadatarepository. In an embodiment, secure TCP/IP links are used for thecommunications among the various components of FIG. 1D, to avoidexposing the data or associated analytics unnecessarily, although notall, or even any, of such precautions are necessary in everyimplementation. The user accesses the controller 115 through the userinterface 110, which may, for example, be implemented on a laptop orother suitable processor-based device. In the illustrated example, thecontroller

115 is hosted on one or more servers 131 within a server farm inNorthern Virginia. For simplicity, the metadata repository 125 is notshown in FIG. 1D, but is also typically hosted on the same server farm,and perhaps the same server, as the controller 115. Those skilled in theart will appreciate that, in many implementations, the repository can belocated elsewhere as long as the communications link between thecontroller and the repository is adequate for the desired throughput.The controller 115 sends metadata instructions to agents 133A-133D, eachof which is typically deployed on a server that is logically if notphysically proximate to an associated data source or data target137A-137D. The data source or data target are typically deployed onserver farms that can be located far from one another, as suggested bythe examples shown in FIG. 1D, where one data source is shown as MexicoCity, another as Johannesburg, a data source/target is shown as SanFrancisco, and finally another data source/target shown as Munich. Theability to manage such geographically diverse sources and targets is adesirable feature of at least some embodiments of the present invention.

As discussed in greater detail hereinafter, the metadata instructionsfrom the controller cause the agent applications to execute E/L/Tfunctions on the data at associated data platform. In the exampleillustrated in FIG. 1D, agents 133A and 133C each send “extract”instructions to associated data platforms 137A and 137C, while agents133B and 133D send “load” and “transform” instructions to associateddata platforms 137B and 137D. Each of the agents 137A-D responds to thecontroller with a status report comprising that agent's response to theE/L/T instruction it received from the controller. It will beappreciated by those skilled in the art that each of the data platforms137A-137D, where each data platform can serve as a data source, a datatarget, or both, depending on what data is to be integrated and to whatpurpose. It will also be appreciated that the agents can send datadirectly to another agent, thus reducing latency in data transmissionsand avoiding bottlenecks at the controller. In addition to using secureTCP/IP connections, in at least some embodiments the agents encrypt andcompress the data being extracted from a data platform and sent out, ordecrypt and decompress the information being received from elsewhere andforwarded to their associated data platform.

FIG. 2 shows an exemplary embodiment of the network hardwareconfiguration on which the data integration system of the presentinvention. Client access is provided via a browser-compatible terminal200 that hosts the User Interface module 105 and can be, for example, apersonal computer running any convenient browser such as InternetExplorer 11.x or later, Google Chrome, Mozilla Firefox, etc., with a CPUconfiguration such as, for example, a pair of Intel Xeon CPU ES-2620processors, or other dual core or better, with four or more gigabytes ofmemory, and an internet-capable network connection. The client accessterminal 200 communicates with a controller 205 via HTTP/HTTPS or otherconvenient protocol. The controller 205 hosts the controller module 110and command line interface module 115, and, in some embodiments, alsohosts at least one agent application and can also host an embeddedrepository. The hardware configuration of the controller 205 can be, forexample, a server having a pair of Intel Xeon ES-2620 processors, dualcore or higher, with four gigabytes of RAM or more, and at least tengigabytes of disk storage together with a gigabit network connection.The operating system for the controller can be any suitable OS, such asLinux, RHEL, Oracle Linux, or Centos Linux 6.1 or later along with Java1.8x or later, Tomcat 7.0.37 or later, and AngularJS 1.x or later.

In some embodiments, agents are embedded in the controller 205, while inother embodiments the agents are hosted on their own systems 210 similarin configuration to the controller although preferably with sixgigabytes or more of memory, and in some cases Microsoft Windows is anacceptable operating system. In at least some embodiments, the agentscommunicate with the controller over a secure TCP/IP connection. Ametadata repository 215 can be embedded in the controller system, or canbe a separate device coupled to the controller in any convenient mannersuch as secure TCP/IP. Typical storage requirements for at least someembodiments can be two gigabytes or greater, and database software canbe Oracle 11.x or Postgres 9.x and later versions or other suitableprograms. A user 220 interacts with the system through the client accessterminal 200 and the command line interface 120 hosted on the controller205.

FIGS. 3A-3B illustrate in greater detail the software components of thedata integration suite of the present invention. As noted previously,these components operate on and control the network hardware of FIG. 2.The User Interface 110 (FIG. 1) can be seen to comprise three functionalareas: Application Administration 300, Data Management 305, and JobMonitoring 310. In an embodiment, Application Administration comprisesthe tasks of managing users, user access, user security, softwarelicense, project, agent configuration and controller and agent logconfiguration, as well as the agent management console. Data Managementcomprises managing user-defined connections, data objects (tables, JSON,XML and file definitions), as well as building the metadata rules forthe extract, load and transform functions performed on the data. Inaddition, data management comprises workflow management, workflowscheduling, impact analysis and maintaining data lineage. The DataManagement module provides an interactive design capability that enablesdebugging of the metadata while design rules for handling theintegration of the data, as described in greater detail hereinafter. TheJob Monitoring module manages job flow monitoring, run and re-start ofjobs, debugging of failed jobs, operational statistics and anoperational dashboard.

The command line interface module 120 permits the user 220 to entercommands relating to job flow, import and export of metadata, and backupand recovery of metadata. Both the User Interface 110 and the CommandLine Interface are accessed by user 220.

The controller 115 comprises three major groups of functionality: aService Manager module, a Rule Compilation Engine, and an ExecutionEngine. The Service Manager manages requests from all clients includingbrowser modules, the CLI, the Scheduler and the Agents. It also managesmetadata requests from the client modules, which can take the form ofdelete/update/insert. Further, the controller manages validation ofuser-defined rules and provides guidance on the design rules. Stillfurther, the controller provides metadata search functionality andbuilds metadata assets as required for providing data lineage. Thecontroller receives commands from the CLI 120, and exchanges metadatainstructions with the User Interface 110.

The Rule Compilation Engine 320 of the Controller 115 preparesplatform-specific instructions for implementing the extract andtransform rules specified by the user. It also manages unit levelplatform instructions and serves requests from the Service Managermodule 315 and the Execution Engine module 325.

The Metadata Repository and Rules Store 125 stores metadata for users,security, projects and agents configurations, as well as data objectmetadata, rules for Extract, Load and Transform functions. In addition,the repository 125 stores platform-specific compiled instructions,operational statistics, and log information. The repository exchangesdata with and receives instructions from the controller 115, includingdesign and compilation metadata as well as operational metadata.

The controller 115 also exchanges metadata instructions with one or moreAgents 330, shown in FIG. 3B (connecting points A and B). In addition,data can flow back from the agents to the controller in instances wherea user requests to view the data at a client. The agents areapplications that receive metadata instructions from the controller. Theinstructions can cause one or more agents either: (1) to extract datafrom an associated source or sources 335, in accordance with the rulesspecified; (2) transfer extracted data directly to a target platform 340or else to another agent; (3) provide to either or both of the sourceand the target platform transform instructions as received from thecontroller 115. In at least some embodiments, and depending upon therules received from the controller, the data is encrypted and compressedbefore sending to another agent. The data sources 335 comprise one ormore of cloud source, databases, files, and so on as shown in FIGS.1A-1B, and can take any suitable form. The target platform 340 istypically a massively parallel processing platform, or processing engineas shown at 150 in FIG. 1A, such as Hadoop, Spark, Teradata, Netezza,Greenplum, Amazon Redshift, Apache Ignite, Oracle, Exadata, and othersimilar platforms.

With the foregoing general description of the components that comprisethe data integration suite and its operating environment, the operationof the invention to achieve data integration can be better appreciated.As shown beginning with FIG. 4, in an embodiment the process ofintegrating large volumes of heterogeneous data from multiple sources inaccordance with the invention begins with a user 400 logging into thesystem via UI 405 and creating a metadata user request, shown at 410.The metadata requests can vary in purpose, depending on the objective ofthe metadata. Typical metadata comprises selection of source and targetdata points, data objects, data flows, job flows, and schedules. Ingeneral, for a new project, the first steps will involve choosing one ormore data points; that is, choosing one or more sources of data forintegration.

If the Request 410 is to create new metadata, the process branches at420 and, as shown at 425, the user selects the type of data point fromthe choices available. The process advances to 430, where a data pointis created for the specific source chosen. In an embodiment, the datapoint comprises several fields, including data point name, type, host,port, agent, and other properties. Once the data point is sufficientlycharacterized, i.e., at least the key fields are populated, theconnection to the controller 115 is tested. In some embodiments allfields must be populated. The controller forwards the connection test tothe selected agent 440, which in turn attempts to link (445) to thesource and target data platforms, shown collectively at 340, designatedin the data point fields. Tests can be conducted for both sources andtargets. If the test succeeds in linking to the source and the target,the data point is considered valid and a save instruction 450 is sent tothe controller 115, which in turn sends the now-successful data point tothe metadata repository 125 together with a save instruction, asindicated at 445.

If the user's objective is to modify a data point, rather than createone, the user enters the data point name at 410 and the process branchesas shown at 420 to an editor step 460. The previously-saved data pointis retrieved from the repository 125 by means of a retrieve message 465to the controller 115 and a retrieve instruction 470 sent to therepository 125, such that the data point is ultimately displayed at theclient terminal or other suitable device, for example a smart phone, forthe user to edit. Once the editing is complete, the edited data point istested as shown at 475, 440 and 445. If the tests are successful, theedited data point can be saved by a message at 465 and a saveinstruction to the repository 125. If multiple data points are to becreated or modified, the process is repeated iteratively as long asnecessary.

Continuing the development of integration metadata, following thesuccessful creation of a data point within the system of the presentinvention, the user typically selects one or more data objects. Oneembodiment of such a process is shown in FIG. 5. As with FIG. 4, theuser logs in and initiates a Create/Modify Metadata User Request, thistime for a data object as selected at

415. At 420, to create a new data object the process branches tomanually choose the type of data object to create from among the varioussources, shown at 500, where the choice is typically the same as thedata point. The newly-created data object comprises a plurality ofattribute and property fields to be populated in order to fullycharacterize the object. The data object attributes and properties cancomprise the data object name, the data object type, and such otherattributes and properties as appropriate for the particular integration.When complete, a “save” message 510 is sent to the controller whichinitiates a “save data object” instruction as shown at 513.

In some instances, such as modifying a data object, it will beappropriate to import the metadata for an existing data point, as shownat 515 which forms the second branch of 420. The metadata is thenretrieved from the source or target data platform 340, or, forfile-based sources/targets, a schema can be provided from a client,shown at 520. The retrieved schema/table/file metadata is provided tothe appropriate agent 330, and then to the controller 115 as shown at525. As shown at 530, the controller forwards a list of data points to aselection step 535, and the process advances to step 540 where the userselects a Tables/Files/Schema definition for file-based data objects.The controller provides at least the relevant Schema/Table/File metadatato enable the selection, shown at 545. The data points list is typicallyprovided by the repository 125 in response to an instruction from thecontroller 115, shown at 550 and 555, respectively.

Once the selection is made, the process advances to step 505 where theuser is able to modify or create (depending on context) the data object.If a specific data object is being modified, it is first retrieved fromthe repository 125 by means of an instruction from the controller 115and a retrieval message 560. When the attributes and properties of theselected data object are sufficiently if not fully populated to permit asave, the user can cause a save message to be sent to the controller inwhich case the data object is saved to the repository as shown at 513.If multiple data objects are to be created or modified, the process isrepeated iteratively as long as necessary.

Following the selection of at least one data point and an associateddata object, creating a data flow is a logical next step. An embodimentof such a process in accordance with the invention is shown in FIG. 6,which also illustrates editing an existing data flow. To create a newdata flow, the user logs in a selects Create/Modify Metadata UserRequest 410, selects data flow at 415, and selects create at 420. Thisadvances to step 600, where the user selects a data point that supportsnative processing from among those available. Typical choices includeHadoop, Netezza, Teradata, and so on. Available data points 605 areretrieved from the repository 125 via instructions from the controller115 in the same manner as described above.

Using Hadoop for purposes of example, once the data objects are selectedfor both source and target, the process advances to step 610 where adata flow is either created or modified. Creating a workflow is selectedfirst for purposes of example. As shown at 615, for each data flow theuser provides the source, transformation rules, and load to target rulesbased on the user's business requirements. An instance of each selectedsource data object, shown at 620A-B, is presented to the user, whoselects operations from those available within that native processingenvironment, as shown at 625. For the example shown in FIG. 6, twooperations, Joiner and Aggregation, are selected as indicated at 630 and635, respectively. Based on the selection of the data point at 600, alist of functions available for that platform is displayed on anexpressions editor for the user's selection, indicated at 640. Theoutput of the sequential transformations 630 and 635 is then provided,along with data objects 605, to a target data object instance, shown at645. Once the functions for each operation on each data object areselected, the data flow can be saved in a manner similar to thatdiscussed for data points and data objects, and as indicated at 650. Thedata flow being developed is in the memory unless it is saved, at whichtime the details are stored in the metadata repository. Modification ofan existing dataflow operates in a very similar way to the modificationof data points and data objects, including particularly the retrieval ofone or more data flows from the repository 125.

With the foregoing discussion of data points, data objects and dataflows in mind, the interactive software components of the presentinvention can be appreciated. In particular, one important aspect of thepresent invention is the ability for the user developing the integrationto see a visual display on the client device of the result achieved byeach applied transformation. In an interactive development with thepresent invention, as the data flow is being developed, the data windowof any transformation shows the preview of the transformed data. Thestate diagram of FIG. 7 illustrates an embodiment of that interaction. Abrowser UI 110 displays a data flow 700 in a data window 705. An objectof data flow 710 is stored in browser memory 715 along with variablesreflecting the last update timestamp 720 and the last compile timestamp725 of the data flow, all as indicated at 730. Thereafter, the browsersubmits REST calls to send the data flow object generated at thatinstant of time when the data window 705 is activated for thetransformation to the controller 115. The browser also sends thetransformation ID 735 for which the data window is active, all indicatedat 740.

The compilation engine 745 then compiles the SQLs 750 required tocompute the data for the active transformation and, via the servicelayer 755, submits them to the agent 330, as indicated at 760. Asindicated at 765, the agent 330 submits the SQL's to the native database770, which causes the database to generate a result set. The nativedatabase is typically a Hadoop platform or massively parallel processingplatform, and is where the transformations, designed as part of the dataflow of the present invention, are executed. In the example of FIG. 7,the result set is transmitted to the controller 115, as indicated at775, and then provided to the browser data window for display to theuser, shown at 780.

Stated more generally, when the data flow needs to be executed from ajob flow, the controller 115 extracts the data stored for that data flowfrom the metadata and sends the data flow object to the compiler engine745. The compiler engine prepares the SQL in accordance with thesemantics of the native platform. Based on the dependencies between thetransformations of the data flow, the compiled SQLs are submitted inthat order of dependency to the agent 330, which then submits the SQL'sto the native platform for execution.

The interactive features of FIG. 7 combine with the data flowcharacteristics of FIG. 6 to illustrate how the present inventionprovides to the user a preview of the data undergoing transformation ateach step of the transformation. This can be better appreciated fromFIG. 8A, which depicts in state diagram form the interactive design ofdata flow of the present invention. The data flow of FIG. 8A is similarto that shown in FIG. 6, and like numerals have been used for likesubstantially similar elements. FIG. 8A differs from FIG. 6 in part withthe addition of a Unit Data Viewer 805, which coordinates viewing thedata in each transformation unit 620A-B, 630, 635 and 645, and Unit DataViewers 810-830, which correspond in order to their respectivetransformation unit. Then, FIG. 8B, which connects to FIG. 8A at point Efrom data flow module 610, provides the structure by which, during aninteractive development, as the data flow is being developed, the datawindow of any transformation shows the preview of the transformed data.By providing such visual and immediate feedback, a developer can quicklytell whether a transformation achieved the intended result. Likewise, afailed result is immediately obvious, and the developer can focusdirectly on the transformation that caused the failure, rather than theprior art approach of running the transformations in what is effectively‘batch’ mode, and then having to review each transformation manuallyuntil the defect is found.

The operation of an embodiment of the data viewer in accordance with theinvention, as shown in FIG. 8B, begins at 835, after which a check ismade at 840 to determine whether any upstream units exist. If yes, acheck is made at 845 to determine whether any current or upstream unitshave been modified since the last timestamp. If yes again, the upstreamunit(s) are compiled at 850 after which the upstream and current unitsare executed in order, 855 and 860, to create staged data files/tablesthat accurately represent normal operation of the transformations. Onceall of the upstream units and the current unit have been compiled andexecuted, the process exits. The compile and execute functions areperformed under the direction of the controller, which receives unitcompile and unit execution requests from the data viewer and in turnsends instructions to the source and target agents, indicated at 865A-C,respectively, to cause the extraction of the relevant data from theSources indicated at 870A-B and the loading of the relevant data to thetarget platform 875. If there are no upstream units, the check at 840will yield a NO, and the process then queries at 875 whether the currentunit has been modified since the last timestamp. If the answer is yes,the current unit is compiled as shown at 850 and then executed as shownat 860. If the current unit has not been modified, or if the check at845 returned a no, indicating that no upstream units had been modifiedsince the last timestamp, the process advances to 880 where a check ismade to determine if the data for the upstream and current units hasbeen persisted. If yes, then the process exits at 885. If no, theupstream and current units are executed as shown at 855 and 860, inorder, and the process then exits at 885. It can thus be appreciatedthat, for each attempt to view the data during interactive development,the present invention ensures that all transformations are performed inproper sequence, to ensure data integrity. Thus, when the user selectsthe data window for any transformation under development, the data shownreflects the impact of all upstream transformation as well as thetransformation performed by the current unit.

Depending upon the embodiment, on clicking the data window for atransformation, the compiler engine is sent with the data flow object atthat instance from the browser memory. For that transformation, thecompiler engine generates the SQL and sends it to the agent forexecution. If data persistence for transformation is enabled then, theresult from the SQL is stored in a table in the native processingplatform's database referred to as the TFORM (transform) database. Thedatabase choice is for the user. The result data in the UI is producedfrom running a select query on the transform database table. If datapersistence is not enabled, the result of the transformation query isrendered directly to UI without storing into a table. The last executiontime of the SQL is stored in memory for each unit. On clicking datawindow, the time of last change to the transformation and the executiontime is compared. If the time of last change is later to the executiontime, the SQL query is executed again to produce the updated data as perthe change. When switching between one transformation window to another,if the data window was already populated with data from an earlierpreparation of preview, the same is stored in memory to avoid replayingthe same query to the data platform again. This is only when there is nochange in the transformation. When populating the data window from asource in data flow, if the source is external to the native platformthe data is extracted from the source platform and stored in a Transform(TFORM) table stored in the native platform's database.

Referring next to FIGS. 9 and 10A-10C, the design and execution of a jobflow can be better appreciated. Starting with FIG. 9, which is a statediagram describing the metadata design of a job flow, the process startsin a manner similar to selection of data points, data objects, and dataflow, as shown at 400, 405, 410, 415 and 420. That advances the processto 900, which provides the option to create or modify a job flow, andincorporates user-specified source(s), transformation rules, and load totarget rules based on business requirements. Data flows are retrievedfrom the repository 125 via the controller 115 and supplied to a list ofchoices, indicated at 910. The selected data flows 915 and 920 are addedto the Job Flow module 900. Then, available job types are selected froma list indicated at 925. Depending on the job type, the selected job iseither performed on success of a prior job, failure of a prior job, orregardless of the success/failure of the prior job. The jobs may beperformed sequentially on different sources, as shown at 915 and 920where 915 uses Teradata as a source while 920 uses Hadoop as a source.For example, the job indicated at 930 is shown as proceeding on a Linuxprocess platform regardless of the success/failure of the job indicatedat 915. Likewise, the job 935 proceeds on the output of jobs 920 and 930regardless of their respective outcomes. The dataset generated by thejob flow can be saved in the native database.

FIGS. 10A-10C show in state diagram form an embodiment of the run-timeexecution of a job flow developed in accordance with FIG. 9. FIG. 10Aillustrates more generally the execution of a job flow across multipleagents 330A-330C and multiple data sources 337A-337C. As seen in FIG.10A, the user initiates a job flow execution at 1010 by instructing thecontroller 115 to retrieve a specific job flow, shown at 1015, from themetadata repository 125. At 1020 a check is made to determine whethereither the job or any dependent objects have been modified after theprevious execution. If so, the job re-compiles as shown at 1023, and themetadata repository 125 is updated accordingly with the new design &compilation metadata. If there have been no modifications since theprevious execution, the process advances to step 1000, a list ofretrieved job flow details. For each job, the controller then createsone or more task threads as shown at 1003, with the number beingappropriate for the number of units within the job. The tasks are thenqueued as shown at 1005.

For each successive task, Task1 through Taskn, the controller issuesmetadata E/L/T instructions to one or more of agents 330A-330C asappropriate for the particular task. The agents then execute theirspecific E/L/T functions by issuing corresponding E/L/T instructions totheir associated source data/targets. Thus, agent1 330B can be seen asissuing Extract instructions to the source/native platform deployed onservers 337C, which causes data to be retrieved from the source andtransmitted back to the agent. The data is then encrypted and compressedby the agent application, and appropriate portions of the encrypted andcompressed data are transmitted directly to agents 330A and 330C. Agent1330A receives its portion of the encrypted/compressed data and decryptsand decompresses it in accordance with the controller's metadatainstructions. Agent1 then transmits the decrypted and decompressed datato the source/native platform on servers 337B and also issues E/L/Tinstructions as directed by the metadata instructions from thecontroller 115. The source/native platform performs the E/L/T functionsas directed, and responds back to agent1 with the appropriate data.

Agent1 330A then encrypts and compresses the responsive data from theplatform on servers 337B, and forwards that encrypted and compresseddata to Agent3, 330C. Agent3 330C, in accordance with metadatainstructions received from controller 115 and now in receipt of datafrom both Agent1 330A and Agent2 330B, decrypts and decompresses thedata and forwards it to the native platform resident on servers 337B,together with Transform instructions. The native platform on servers337B perform the transform as directed, and the task completes. Asappropriate, status updates for the jobflow, job and unit are stored onthe metadata repository 125. The process the repeats for the next taskuntil the last task is of a given Task Thread is completed, after whichthe process repeats for each additional Task Thread. Finally, after thelast task for the last Task Thread in a given job is completed, theprocess advances to the next job. The operational metadata for the jobflow, jobs, and units is updated in the repository 125 as shown at 1027.

FIGS. 10B and 10C can be seen to connect at points R, S, T, U, V and W.In particular, FIGS. 10A and 10B-10C illustrate in greater detail Jobflow/Job/Unit threads, and their interdependences that must be takeninto account for proper execution. Thus, controller 115 comprises jobflow details block 1000 and Thread Queue 1005. Job Flow Details Blockshows jobs 915, 920, 930 and 935, also indicated as Jobs J1, J2, J3 andJ4, respectively. Job flow execution is initiated from the clientmodule, indicated at 1010. Upon initiation, specific job flow detailsare retrieved from the metadata repository 125, as indicated at 1015. Acheck is made at 1020 to determine if the job or any dependent objectshave been modified since previous execution. If so, the job flow iscompiled and brought current, as indicated at 1023, and the results arereported to the Job Flow Details block 1000. In addition, the compiled,updated jobflow design metadata is stored in the repository 125, fromwhich it can be retrieved in the future for iterations where no interimchanges have been made.

The Thread Queue 1005 illustrates an example of the interdependenciesthat can occur among various jobs. Thus, beginning with Job Flow 1indicated at 1025, the thread queue first performs job J1, whichcomprises a single Unitl. Upon completion, the output of J1 is providedto J2 and J2 initiated only if J1 was successful. However, the output ofJ1 is provided to J3 whether J1 was successful or not. J3 is, again asingle unit, and the output of J3 is provided to J4, and J4 initiated,regardless whether J3 is successful or not.

However, J2, which is initialized only if J1 ended successfully,comprises five units with significant interdependencies. Unit 1 of J2,indicated as 920A and also as J2-U1, provides its output to J2-U3, or920C, but J2-U3 initiates only upon success of J2-U1. Likewise, J2-U2provides its output to J2-U3, but J2-U3 only initiates if J2-U2 issuccessful. J2-U4 receives the output of J2-U3, but initiates only ifJ2-U3 was successful. J2-U4 provides its output to J2-U5, but J2-U5 onlyinitiates if J2-U4 ends in success. However, the final output of J2 isprovided to J4 regardless of success or failure. The process ends onceJob J4 completes, and the updated operational metadata associated withthe job flow/job/unit is stored in the repository 125 as shown at 1027.It will be appreciated that FIG. 10B illustrates the agents 330A-Cassociated with each of the relevant Sources 340A-C. As with the priorfigures discussed hereinabove, the agents receive instructions inconnection with each related job, and redirect those instructions to therelevant Sources for execution. Once the data is returned, the agentspass the results back to the controller for further processing.

Referring next to FIG. 11, the relationship between the controller 115and an agent 330 can be better appreciated. As noted previously, thecontroller 115 is the main software application of the data integrationsuite of the present invention. In a typical arrangement, and as notedin connection with FIG. 1C, the controller 115 is installed on a serverrunning a Linux operating system. Further, the controller 115 typicallycomprises some form of Java Message Oriented Middleware API, such asApache MQ, as the Java Messenger Service provider running in thecontroller's host. In an embodiment, the active message queue ActiveMQ,indicated at 1105, comprises a status queue 1110, an agent_name queue1115 where there is one queue per agent name associated with thecontroller, and a ping queue 1120. The agent queue produces objects andtext messages based on instructions and messages from the controller.The objects are the compiled code that is to be executed on the nativedatabase.

Agent applications, indicated at 1150, can also be hosted on a Linuxserver or other suitable server such as Windows, indicated at 1155. Theagent_name listener 1160 listens to the agent_name queue 1115 for itsname and receives messages which are then executed against the nativedatabase. The status of the executions and results of the executions areproduced to the status queue 1110. The status listener component 1125listens to the status queue 1110 and receives any message sent by anassociated agent. The ping queue 1120 receives each agent's heart beatmessage, typically sent every minute, to ensure up to date informationabout the health of each agent. The heart beat listener 1130 in thecontroller receives the same to know whether the agent is active. Anagent status pool service 1135 runs in the controller to maintain thestatus of all agents associated with the controller.

Referring next to FIG. 12A-12B, an embodiment of agent-to-agent datatransfers can be better understood, including associated compression andencryption features. FIGS. 12A and 12B connect at the points indicatedas J, K, and L. The source establishes connection to the target hostusing a data receiving port. The metadata of the data transfer process,such as the data file/pipe to be created in the target, and theencryption and compression algorithm to be used, are sent to the targetagent, shown at 1250. The target acknowledges after setting up the datareceiver 1253, and creates a named pipe as shown at 1255 including adata file name. As discussed in more detail below, once acknowledged bythe target, the source starts sending chunks of data over the network tothe data receiver in the target. The target starts receiving the data,chunk by chunk, and writes the data to a target file. If encryption orcompression is on, the data is first written to a staging file which isthe decompressed and decrypted, before writing to a target file. Oncethe last chunk of data is received, the Target acknowledges completionof the data transfer and closes its resources. Upon receiving theacknowledgement, the source closes its resources.

Referring more specifically to FIGS. 12A-12B, the details of the processdescribed above can be better appreciated. The process starts at 1200,with a user's instruction of a job flow execution to the controller 115.The controller retrieves from the metadata repository 125 the specificunit execution request, which can be specific to unit/type and comprisesextract, load or transform metadata as specified in the job flowdevelopment phase described above. The specific unit execution requestis then routed through a check, 1205, which direct the appropriatemetadata to one or more agents, shown as two in the illustrated example.In the example, agent1 1210 is associated with a source and receivesextract/load instructions, while agent2 1215 is associated with a targetand receives transform instructions which are directed to a transformunit 1217.

The source agent 1210 starts unit execution as shown at 1203, anddirects an Extract Data unit 1207 to issue extract metadata instructionsto source platforms 1209A-1209B. The responsive data is then transmittedback to the Extract Data unit 1207, after which determinations whetherto compress and encrypt the data are made as shown at 1211 and 1213,with the associated compression and encryption units shown at 1217 and1219. Compression and encryption need not be used in every embodiment,but provide additional security when used. If the encryption option isturned on, the data is encrypted based on an algorithm chosen by theuser, as, for example, Payload encryption with Java Advanced EncryptionStandard (“AES”), and support for 128, 192 and 256 bit encryption. Ifcompression is turned on the data is compressed based on the user'schoice of suitable codecs including, for example, the zlib or gzipcodecs. After compression and encryption have been completed, or if nocompression or encryption is required, the data is forwarded to a DataPrep unit 1221. If necessary to divide up the data for efficienttransmission, the Data Prep unit divides the data file into chunkssuitable for transmission on the data pipe that has been establishedbetween the source agent and the target agent, typically secure TCP/IPas shown in FIG. 1D.

As noted above, on start-up of target agent 1215, a data receiver socketservice such as Netty is also started listening on a port, as shown at1253. Any number of source agents 1205 can start communicating to thetarget agent on this socket. The creation of the named pipe at 1255causes the target agent 1215 to send to a Send Data to Load Agent unit1223 on the source agent 1210 a message acknowledging that the target isready to receive data.

Once acknowledged by the target, the source starts sending chunks ofdata over the network to the data receiver in the target. The processinitially performs a check at 1225 to determine if the extract agent isthe same at the load agent. If not, the process advances to block 1223,but, if it is, the process loads data to the target platform as shown at1227 by the issuance of load instructions to write to the target, shownin FIG. 12B. Block 1223 directs the data to the Encrypt block in thetarget 1215. The target 1215 starts receiving the data, chunk by chunk,and writes the data to a target file 1260. If the source data has beenencrypted or compressed, or both, as advised by block 1250, the data isfirst written to a staging file 1265 and is the decompressed anddecrypted, 1270-1275, before writing to the target file 1260. Once thelast chunk of data is received, the Target sends to the Source anacknowledgement 1280 of completion of the data transfer and closes itsresources. Upon receiving the acknowledgement, the source closes itsresources, 1285.

Referring next to FIG. 13, an embodiment of an agent management console1300 in accordance with the invention can be better appreciated. Theagent management console is a component of the user interface, and theoperations to support it are provided by the Controller 115. Asdiscussed in connection with FIG. 11, the Controller 115 comprises aping queue 1120 to which each agent associated with that controllerreports its availability. In addition, the controller comprises a statusqueue 1110. Each agent associated with a particular controller generatesevents with the status of the jobs it is assigned, and produces them tothe status queue 1110. Through the status listeners 1125 and heart beatlisteners 1130 running in the controller, the agent management consolecomponent obtains the heart beat information for each of the agentsassociated with it. It also obtains the report on the number ofprocesses active in each agents, number of jobs processed in a timewindow, the disk usage, instantaneous memory usage in the agentenvironment by the agent process, the CPU utilization by the agentprocess, which allows efficient load balancing.

Referring next to FIGS. 14A-14B, which are joined at points M, N, P andQ, a state diagram illustrates the relationship between the controller115 and agent groups. The process starts at 1400, with the start of aunit, and a query determines whether a unit is associated with an agentor an agent group. If the unit is associated with an agent, the agentstatus from agent groups 330A and 330B (FIG. 14B) is retrieved at 1410.If not, an agent is assigned, step 1415. If the unit is already assignedto an agent, the controller identifies the active agent with the leastactive threads, 1420 and may reassign the unit to the agent to achieveload balancing. Once the unit has been associated with an agent, theprocess completes at 1425.

FIG. 15 is a block diagram illustrating an example wired or wirelesssystem 1550 that may be used in connection with various embodimentsdescribed herein. For example the system 1550 may be used as or inconjunction with a data system (such as a data source or a data target),an agent, a controller, a metadata/rules engine, a server farm, or anyother processing device or machine as previously described herein. Thesystem 1550 can be a conventional personal computer, computer server,personal digital assistant, smart phone, tablet computer, or any otherprocessor enabled device that is capable of wired or wireless datacommunication. Other computer systems and/or architectures may be alsoused, as will be clear to those skilled in the art.

The system 1550 preferably includes one or more processors, such asprocessor 1560. Additional processors may be provided, such as anauxiliary processor to manage input/output, an auxiliary processor toperform floating point mathematical operations, a special-purposemicroprocessor having an architecture suitable for fast execution ofsignal processing algorithms (e.g., digital signal processor), a slaveprocessor subordinate to the main processing system (e.g., back-endprocessor), an additional microprocessor or controller for dual ormultiple processor systems, or a coprocessor. Such auxiliary processorsmay be discrete processors or may be integrated with the processor 1560.

The processor 1560 is preferably connected to a communication bus 1555.The communication bus 1555 may include a data channel for facilitatinginformation transfer between storage and other peripheral components ofthe system 1550. The communication bus 1555 further may provide a set ofsignals used for communication with the processor 1560, including a databus, address bus, and control bus (not shown). The communication bus1555 may comprise any standard or non-standard bus architecture such as,for example, bus architectures compliant with industry standardarchitecture (“ISA”), extended industry standard architecture (“EISA”),Micro Channel Architecture (“MCA”), peripheral component interconnect(“PCI”) local bus, or standards promulgated by the Institute ofElectrical and Electronics Engineers (“IEEE”) including IEEE 488general-purpose interface bus (“GPIB”), IEEE 696/S-100, and the like.

System 1550 preferably includes a main memory 1565 and may also includea secondary memory 1570. The main memory 1565 provides storage ofinstructions and data for programs executing on the processor 1560. Themain memory 1565 is typically semiconductor-based memory such as dynamicrandom access memory (“DRAM”) and/or static random access memory(“SRAM”). Other semiconductor-based memory types include, for example,synchronous dynamic random access memory (“SDRAM”), Rambus dynamicrandom access memory (“RDRAM”), ferroelectric random access memory(“FRAM”), and the like, including read only memory (“ROM”).

The secondary memory 1570 may optionally include a internal memory 1575and/or a removable medium 1580, for example a floppy disk drive, amagnetic tape drive, a compact disc (“CD”) drive, a digital versatiledisc (“DVD”) drive, etc. The removable medium 1580 is read from and/orwritten to in a well-known manner. Removable storage medium 1580 may be,for example, a floppy disk, magnetic tape, CD, DVD, SD card, etc.

The removable storage medium 1580 is a non-transitory computer readablemedium having stored thereon computer executable code (i.e., software)and/or data. The computer software or data stored on the removablestorage medium 1580 is read into the system 1550 for execution by theprocessor 1560.

In alternative embodiments, secondary memory 1570 may include othersimilar means for allowing computer programs or other data orinstructions to be loaded into the system 1550. Such means may include,for example, an external storage medium 1595 and an interface 1570.Examples of external storage medium 1595 may include an external harddisk drive or an external optical drive, or and external magneto-opticaldrive.

Other examples of secondary memory 1570 may include semiconductor-basedmemory such as programmable read-only memory (“PROM”), erasableprogrammable read-only memory (“EPROM”), electrically erasable read-onlymemory (“EEPROM”), or flash memory (block oriented memory similar toEEPROM). Also included are any other removable storage media 1580 andcommunication interface 1590, which allow software and data to betransferred from an external medium 1595 to the system 1550.

System 1550 may also include an input/output (“I/O”) interface 1585. TheI/O interface 1585 facilitates input from and output to externaldevices. For example the I/O interface 1585 may receive input from akeyboard or mouse and may provide output to a display. The I/O interface1585 is capable of facilitating input from and output to variousalternative types of human interface and machine interface devicesalike.

System 1550 may also include a communication interface 1590. Thecommunication interface 1590 allows software and data to be transferredbetween system 1550 and external devices (e.g. printers), networks, orinformation sources. For example, computer software or executable codemay be transferred to system 1550 from a network server viacommunication interface 1590. Examples of communication interface 1590include a modem, a network interface card (“NIC”), a wireless data card,a communications port, a PCMCIA slot and card, an infrared interface,and an IEEE 1394 fire-wire, just to name a few.

Communication interface 1590 preferably implements industry promulgatedprotocol standards, such as Ethernet IEEE 802 standards, Fiber Channel,digital subscriber line (“DSL”), asynchronous digital subscriber line(“ADSL”), frame relay, asynchronous transfer mode (“ATM”), integrateddigital services network (“ISDN”), personal communications services(“PCS”), transmission control protocol/Internet protocol (“TCP/IP”),serial line Internet protocol/point to point protocol (“SLIP/PPP”), andso on, but may also implement customized or non-standard interfaceprotocols as well.

Software and data transferred via communication interface 1590 aregenerally in the form of electrical communication signals 1605. Thesesignals 1605 are preferably provided to communication interface 1590 viaa communication channel 1600. In one embodiment, the communicationchannel 1600 may be a wired or wireless network, or any variety of othercommunication links. Communication channel 1600 carries signals 1605 andcan be implemented using a variety of wired or wireless communicationmeans including wire or cable, fiber optics, conventional phone line,cellular phone link, wireless data communication link, radio frequency(“RF”) link, or infrared link, just to name a few.

Computer executable code (i.e., computer programs or software) is storedin the main memory 1565 and/or the secondary memory 1570. Computerprograms can also be received via communication interface 1590 andstored in the main memory 1565 and/or the secondary memory 1570. Suchcomputer programs, when executed, enable the system 1550 to perform thevarious functions of the present invention as previously described.

In this description, the term “computer readable medium” is used torefer to any non-transitory computer readable storage media used toprovide computer executable code (e.g., software and computer programs)to the system 1550. Examples of these media include main memory 1565,secondary memory 1570 (including internal memory 1575, removable medium1580, and external storage medium 1595), and any peripheral devicecommunicatively coupled with communication interface 1590 (including anetwork information server or other network device). Thesenon-transitory computer readable mediums are means for providingexecutable code, programming instructions, and software to the system1550.

In an embodiment that is implemented using software, the software may bestored on a computer readable medium and loaded into the system 1550 byway of removable medium 1580, I/O interface 1585, or communicationinterface 1590. In such an embodiment, the software is loaded into thesystem 1550 in the form of electrical communication signals 1605. Thesoftware, when executed by the processor 1560, preferably causes theprocessor 1560 to perform the inventive features and functionspreviously described herein.

The system 1550 also includes optional wireless communication componentsthat facilitate wireless communication over a voice and over a datanetwork. The wireless communication components comprise an antennasystem 1610, a radio system 1615 and a baseband system 1620. In thesystem 1550, radio frequency (“RF”) signals are transmitted and receivedover the air by the antenna system 1610 under the management of theradio system 1615.

In one embodiment, the antenna system 1610 may comprise one or moreantennae and one or more multiplexors (not shown) that perform aswitching function to provide the antenna system 1610 with transmit andreceive signal paths. In the receive path, received RF signals can becoupled from a multiplexor to a low noise amplifier (not shown) thatamplifies the received RF signal and sends the amplified signal to theradio system 1615.

In alternative embodiments, the radio system 1615 may comprise one ormore radios that are configured to communicate over various frequencies.In one embodiment, the radio system 1615 may combine a demodulator (notshown) and modulator (not shown) in one integrated circuit (“IC”). Thedemodulator and modulator can also be separate components. In theincoming path, the demodulator strips away the RF carrier signal leavinga baseband receive audio signal, which is sent from the radio system1615 to the baseband system 1620.

If the received signal contains audio information, then baseband system1620 decodes the signal and converts it to an analog signal. Then thesignal is amplified and sent to a speaker. The baseband system 1620 alsoreceives analog audio signals from a microphone. These analog audiosignals are converted to digital signals and encoded by the basebandsystem 1620. The baseband system 1620 also codes the digital signals fortransmission and generates a baseband transmit audio signal that isrouted to the modulator portion of the radio system 1615. The modulatormixes the baseband transmit audio signal with an RF carrier signalgenerating an RF transmit signal that is routed to the antenna systemand may pass through a power amplifier (not shown). The power amplifieramplifies the RF transmit signal and routes it to the antenna system1610 where the signal is switched to the antenna port for transmission.

The baseband system 1620 is also communicatively coupled with theprocessor 1560. The central processing unit 1560 has access to datastorage areas 1565 and 1570. The central processing unit 1560 ispreferably configured to execute instructions (i.e., computer programsor software) that can be stored in the memory 1565 or the secondarymemory 1570. Computer programs can also be received from the basebandprocessor 1610 and stored in the data storage area 1565 or in secondarymemory 1570, or executed upon receipt. Such computer programs, whenexecuted, enable the system 1550 to perform the various functions of thepresent invention as previously described. For example, data storageareas 1565 may include various software modules (not shown) that areexecutable by processor 1560.

Various embodiments may also be implemented primarily in hardware using,for example, components such as application specific integrated circuits(“ASICs”), or field programmable gate arrays (“FPGAs”). Implementationof a hardware state machine capable of performing the functionsdescribed herein will also be apparent to those skilled in the relevantart. Various embodiments may also be implemented using a combination ofboth hardware and software.

Furthermore, those of skill in the art will appreciate that the variousillustrative logical blocks, modules, circuits, and method stepsdescribed in connection with the above described figures and theembodiments disclosed herein can often be implemented as electronichardware, computer software, or combinations of both. To clearlyillustrate this interchangeability of hardware and software, variousillustrative components, blocks, modules, circuits, and steps have beendescribed above generally in terms of their functionality. Whether suchfunctionality is implemented as hardware or software depends upon theparticular application and design constraints imposed on the overallsystem. Skilled persons can implement the described functionality invarying ways for each particular application, but such implementationdecisions should not be interpreted as causing a departure from thescope of the invention. In addition, the grouping of functions within amodule, block, circuit or step is for ease of description. Specificfunctions or steps can be moved from one module, block or circuit toanother without departing from the invention.

Moreover, the various illustrative logical blocks, modules, and methodsdescribed in connection with the embodiments disclosed herein can beimplemented or performed with a general purpose processor, a digitalsignal processor (“DSP”), an ASIC, FPGA or other programmable logicdevice, discrete gate or transistor logic, discrete hardware components,or any combination thereof designed to perform the functions describedherein. A general-purpose processor can be a microprocessor, but in thealternative, the processor can be any processor, controller,microcontroller, or state machine. A processor can also be implementedas a combination of computing devices, for example, a combination of aDSP and a microprocessor, a plurality of microprocessors, one or moremicroprocessors in conjunction with a DSP core, or any other suchconfiguration.

Additionally, the steps of a method or algorithm described in connectionwith the embodiments disclosed herein can be embodied directly inhardware, in a software module executed by a processor, or in acombination of the two. A software module can reside in RAM memory,flash memory, ROM memory, EPROM memory, EEPROM memory, registers, harddisk, a removable disk, a CD-ROM, or any other form of storage mediumincluding a network storage medium. An exemplary storage medium can becoupled to the processor such the processor can read information from,and write information to, the storage medium. In the alternative, thestorage medium can be integral to the processor. The processor and thestorage medium can also reside in an ASIC.

The above description of the disclosed embodiments is provided to enableany person skilled in the art to make or use the invention. Variousmodifications to these embodiments will be readily apparent to thoseskilled in the art, and the generic principles described herein can beapplied to other embodiments without departing from the spirit or scopeof the invention. Thus, it is to be understood that the description anddrawings presented herein represent a presently preferred embodiment ofthe invention and are therefore representative of the subject matterwhich is broadly contemplated by the present invention. It is furtherunderstood that the scope of the present invention fully encompassesother embodiments that may become obvious to those skilled in the artand that the scope of the present invention is accordingly not limited.

Having fully described a preferred embodiment of the invention andvarious alternatives, those skilled in the art will recognize, given theteachings herein, that numerous alternatives and equivalents exist whichdo not depart from the invention. It is therefore intended that theinvention not be limited by the foregoing description, but only by theappended claims.

1.-6. (canceled)
 7. A technical system comprising: one or morenon-transitory computer readable mediums configured to store executableprogrammed modules; one or more processors, each of the one or moreprocessors communicatively coupled with at least one of thenon-transitory computer readable mediums, the one or more processorsconfigured to: send first data set extraction instructions to a firstagent by a controller; send first data set transformation instructionsto a second agent by the controller; extract a first data set from afirst data system by the first agent in accordance with the first dataset extraction instructions; send the first data set to the second agentvia a network by the first agent in accordance with the first data setextraction instructions; load the first data set to a second data systemby the second agent; provide the first data set transformationinstructions to the second data system by the second agent; send seconddata set extraction instructions to the second agent by the controller;send second data set transformation instructions to a third agent by thecontroller; extract a second data set from the second data system by thesecond agent in accordance with the second data set extractioninstructions, wherein the second data set corresponds to at least aportion of the first data set; send the second data set to the thirdagent via a network by the second agent in accordance with the seconddata set extraction instructions; load the second data set to a thirddata system by the third agent; and provide the second data settransformation instructions to the third data system by the secondagent.
 8. The system of claim 7, wherein the second data system executesthe first data set transformation instructions on the first data set. 9.The system of claim 7, wherein the first agent is further configured tocompress the extracted first data set prior to sending the first dataset.
 10. The system of claim 7, wherein the first agent is furtherconfigured to encrypt the extracted first data set prior to sending thefirst data set.
 11. The system of claim 7, wherein the first agent isfurther configured to compress and encrypt the extracted first data setprior to sending the first data set.
 12. The system of claim 7, whereinthe first agent is further configured to audit the sending of the firstdata set to the second agent.
 13. The system of claim 7, wherein thethird data system executes the second data set transformationinstructions on the second data set.
 14. The system of claim 7, whereinthe second agent is further configured to compress the extracted seconddata set prior to sending the second data set.
 15. The system of claim7, wherein the second agent is further configured to encrypt theextracted second data set prior to sending the second data set.
 16. Thesystem of claim 7, wherein the second agent is further configured tocompress and encrypt the extracted second data set prior to sending thesecond data set.
 17. The system of claim 7, wherein the second agent isfurther configured to audit the sending of the second data set to thethird agent. 18.-34. (canceled)
 35. A method for viewing intermediatetransformations, comprising: identifying a plurality of transformationrules comprising a first transformation unit; presenting on a userinterface a representation of a first data set; applying a first of theplurality of transformation rules to the first data set to generate afirst modified first data set; presenting on the user interface arepresentation of the first modified data set to be further modified byapplication of a second of the plurality of transformation rules;receiving an instruction to modify the second of the plurality oftransformation rules and generating a modified second of the pluralityof transformation rules; applying the modified second of the pluralityof transformation rules to the first modified first data set to generatea second modified first data set; presenting on the user interface arepresentation of the second modified first data set; receiving aninstruction to finalize the plurality of transformation rules comprisingthe first transformation unit; and finalizing the plurality oftransformation rules comprising the first transformation unit andfinalizing the first transformation unit.
 36. The method of claim 35,wherein the second modified first data set is to be further modified byapplication of a third of the plurality of transformation rules, furthercomprising prior to receiving the instruction to finalize: receiving aninstruction to modify the third of the plurality of transformation rulesand generating a modified third of the plurality of transformationrules; applying the modified third of the plurality of transformationrules to the second modified first data set to generate a third modifiedfirst data set; and presenting on the user interface a representation ofthe third modified first data set.
 37. The method of claim 36, whereinthe third modified first data set is to be further modified byapplication of a fourth of the plurality of transformation rules,further comprising prior to receiving the instruction to finalize:receiving an instruction to modify the fourth of the plurality oftransformation rules and generating a modified fourth of the pluralityof transformation rules; applying the modified fourth of the pluralityof transformation rules to the third modified first data set to generatea fifth modified first data set; and presenting on the user interface arepresentation of the fifth modified first data set. 38.-42. (canceled)43. A technical system comprising: at least one controller apparatuscomprising one or more non-transitory computer readable mediumsconfigured to store executable programmed modules and data and one ormore processors, each of the one or more processors communicativelycoupled with at least one of the non-transitory computer readablemediums; a plurality of first agents communicatively coupled with the atleast one controller apparatus via a network, each first agentcomprising one or more non-transitory computer readable mediumsconfigured to store executable programmed modules and data and one ormore processors, each of the one or more processors communicativelycoupled with at least one of the non-transitory computer readablemediums, wherein each of the plurality of first agents corresponds to atleast one data system; a plurality of second agents communicativelycoupled with the at least one controller apparatus via a network andcommunicatively coupled with at least one of the plurality of firstagents via a network, each second agent comprising one or morenon-transitory computer readable mediums configured to store executableprogrammed modules and data and one or more processors, each of the oneor more processors communicatively coupled with at least one of thenon-transitory computer readable mediums, wherein each of the pluralityof second agents corresponds to at least one data system; wherein thecontroller is further configured to: send first data set extractioninstructions to one or more of the plurality of first agents; and sendfirst data set transformation instructions to one or more of theplurality of second agents; wherein each of the one or more first agentsreceiving first data set extraction instructions from the controller isfurther configured to: extract a first data set from its respectivecorresponding data system in accordance with the first data setextraction instructions; and send the extracted first data set to one ofthe plurality of second agents via a network in accordance with thefirst data set extraction instructions, wherein the second agentreceived first data set transformation instructions from the controller;wherein each of the one or more second agents receiving an extractedfirst data set from a first agent and receiving first data settransformation instructions from the controller is further configuredto: load the first data set to its respective corresponding data system;and provide the first data set transformation instructions to itsrespective corresponding data system.
 44. The system of claim 43,wherein each second agent corresponding data system executes itsrespective first data set transformation instructions on its respectivefirst data set.
 45. The system of claim 43, wherein each of the one ormore first agents is further configured to compress the extracted firstdata set prior to sending the extracted first data set.
 46. The systemof claim 43, wherein each of the one or more first agents is furtherconfigured to encrypt the extracted first data set prior to sending theextracted first data set.
 47. The system of claim 43, wherein each ofthe one or more first agents is further configured to compress andencrypt the extracted first data set prior to sending the extractedfirst data set.
 48. The system of claim 43, wherein each of the one ormore first agents is further configured to audit the sending of theextracted first data set.